Process and system for encoding and playback of stereoscopic video sequences

ABSTRACT

A method for decoding a compressed image stream, the image stream having a plurality of frames, each frame consisting of a merged image including pixels from a left image and pixels from a right image. The method involves the steps of receiving each merged image; changing a clock domain from the original input signal to an internal domain; for each merged image, placing at least two adjacent pixels into an input buffer and interpolating an intermediate pixel, for forming a reconstructed left frame and a reconstructed right frame according to provenance of the adjacent pixels; and reconstructing a stereoscopic image stream from the left and right image frames. The invention also teaches a system for decoding a compressed image stream.

FIELD OF THE INVENTION

The present invention relates generally to a process and system forencoding and decoding a dual program image sequence, and, moreparticularly, to a process and system for compressing two image sequencesignals on a single video signal and decoding said single video signalto reproduce two image sequence programs or a three-dimensionalstereoscopic program in multiple viewing formats. Although the inventionwill be described hereinafter by reference to processing ofthree-dimensional stereoscopic programs such as movies, it should bedeemed to be within the scope of the present invention to apply to theprocessing of any pair of video sequences, regardless of any differencesin the respective video content of each sequence. It should beunderstood that the expressions “decoding” and “decompressing” are usedinterchangeably within the present description, as are the expressions“encoding” and “compressing”.

BRIEF DESCRIPTION OF THE PRIOR ART

Since the invention of the stereoscope in 1947, several systems havebeen developed to enable a viewer to view three-dimensional (3D)programs through the reproduction of a first image sequence intended forviewing by the viewer's left eye and a second sequence of images of thesame scene and at the same time but with a parallax with respect to thefirst image sequence, intended to be viewed exclusively by the viewer'sright eye, thereby replicating the principles of naturalthree-dimensional vision. Since the 1950s, many films have been madeusing dual camera head systems to pick up stereo pairs of images intime-synchronism and with a parallax to enable a viewer at reproductionto perceive the effect of depth, so to provide a more complete andexciting viewing experience.

At present, home theatre systems are rapidly penetrating the householdmarket and very sophisticated and high quality systems are gaining inpopularity, responding to a need for a high quality cinematographicexperience at home. Nevertheless, existing stereoscopic reproductionsystems are still far from fulfilling the expectations of viewers andare still not integrated into even the most advanced home theatresystems available. The reason mostly lies on the relatively poor imagequality (fade colours and/or stair-stepping diagonals) and the fatigueand discomfort caused by the usual flicking and lack of spatial realismIndeed, since two different programs are being presented with equipmentintended for single video program presentation, such as a televisionset, sharing of the technical resources between two video signals leadsto loss of image spatial resolution and flicking due to the reduction byhalf of the frame presentation rate for each eye and contrast betweenimage fields and a black background.

A typical existing stereoscopic reproduction technology consists inencoding the first image sequence information in the even line field ofan interlaced video signal and the information of the second imagesequence in the odd line field of the signal. At playback, shutterspectacles are used to block one of the viewer's eyes duringpresentation of the even lines and the other eye during presentation ofthe odd lines. As normal images comprising even and odd lines aretypically presented in two successive scan periods of 1/60s, each eyesees the stereoscopic program as a sequence of 1/60s images followed by1/60s blackout periods, to enable each eye to view 30 frames per second(fps). Moreover, each reproduced image is constituted by alternatingimage lines and black lines. Obviously, the stereoscopic images soreproduced lose half of their topological information and the 50% dutycycles (both in space and in time) induce loss of brightness andflicking, as confirmed by experience.

A solution to such limitations, shortcomings and drawbacks, is topresent complete stereoscopic images at a rate of at least 60 fps (30full frames per second per eye), which would normally require at leasttwice the signal bandwidth required by a non-stereo (planar) program.Elimination of flicking in a room presenting relatively high contrastbetween the displayed pictures and ambient lighting, further requires avertical scan (and shutter spectacle) frequency of up to 120 Hz, toenable presentation of up to 60 full definition images per second toeach eye. While such a frequency is not widely available, flickerlesspresentation of stereoscopic program can be set up by using two digitalvideo projectors of current manufacture, receiving respectively a firstand a second image sequence of the stereoscopic program at a continuousrate of 30 fps each. The output of each projector is optically filteredto produce a vertically and a horizontally polarized output projectingimages in register and in perfect time synchronism on a special silvercoated screen. Eyewear comprising differently polarized glasses can beworn by a viewer to reveal the three-dimensional effects. Such asolution is obviously very expensive and does not meet marketexpectations for a home theatre system.

However, very fast and relatively affordable projectors using the DLP(Digital Light Processing) technology are now available that couldprovide a presentation rate up to 120 fps, so that a single projectorcould alternatively present images of stereo pair sequences at asufficiently high rate to substantially eliminate flicking even in ahigh contrast environment Also, high-end CRT projectors and computermonitors could provide such a compatible definition and refresh rate.

Nevertheless, a major limitation of such systems remains that mostcurrent standards for storage and broadcast (transport) of video programinformation limit the flow of full frame images to 30 fps, which isapproximately half of the capacity required to store and present a highquality stereoscopic program originally comprised of two 24 (AmericanMotion Picture), 25 (PAL or SECAM) or 30 fps (NTSC Video) programs.Furthermore, since motion picture movies are always captured andrecorded at a rate of 24 frames per second, the dual problem ofcompressing two 24 fps programs into a single 30 fps signal andthereafter expanding such a signal to present the two programs at a rateof 30 to 60 fps each must be addressed. Therefore, the future of the 3Dhome theatre lies on the capacity to encode and decode a stereoscopicvideo signal to comply with standard recorders, players and broadcastequipment of present manufacture treating a 30 fps signal compressed anddecompressed using a protocol such as MPEG-I or MPEG-2 (Moving PictureImage Coding Group) compression/decompression protocol of the MAINprofile (vs MVP), so that negligible loss of information or distortionis induced throughout the process.

A few technologies of the prior art have taught solutions to overcomeone or more of the above-mentioned shortcomings and limitations.Firstly, the 3:2 pull-down compression method can be used to create a 30fps stereoscopic interlaced signal from a 24 fps interlaced picturesequence. With this method the original image sequence is time-expandedby creating and inserting one new picture after every four pictures ofthe original sequence. The new picture comprises the even lines of thepreceding picture in one field and the odd lines of the next picture inits second field. Obviously, each picture of the original program may becomprised of a first field comprising a portion of a left view image anda second field comprising a portion of a right view image of astereoscopic program. A 30 fps stereoscopic program can thereby beobtained from a 24 fps left eye sequence and a 24 fps right eyesequence. With such a technique however, the resulting 30 fps programpresents anachronism and topological distortion due to the combinationin certain pictures of lines belonging to images captured at differenttimes. This yields a poor result, lacking in realism and causing eyefatigue and discomfort to the viewer. When used to present astereoscopic program, this technique further suffers from the samelimitations and drawbacks as discussed hereinabove about the interlacedsignal compression technique.

Furthermore, many stereoscopic display devices have been developed usingdifferent input signals incompatible with one another and requiringdifferent transport (storage or distribution) formats (columninterleaved, row interleaved, simultaneous dual presentation, pageflipping, anaglyphic, etc.). A solution to bring a stereoscopic videoprogram to different systems at the same time while allowing for 2Dviewing would be to simultaneously broadcast or store on severalphysical media in all the existing formats. Obviously, that wouldneither be practical nor economical. Therefore, the future ofstereoscopic video at home requires a stereoscopic video signal and avideo processing apparatus that have the ability to generatemultiple/universal stereoscopic output formats compatible with currentand future stereoscopic display devices while allowing for normal 2Dviewing.

Many patents also teach compression techniques to reduce two 30 fpssignals to be carried through a single channel with a 30 fps capacity,some of them being designed to be transparent for the MPEGcompression/decompression process. However, these techniques do notfeature temporal interpolation as needed to create the missing frames toconvert for instance a 24 fps sequence to 30 fps, or to convert a 30 fpssequence to a 48, 60, 72, 96 or 120 fps sequence, while preserving imagequality and providing a comfortable viewing experience. Furthermore,they do not have the ability to generate multiple stereoscopic outputformats from the same video signal and video processing apparatus.

For instance, U.S. Pat. No. 5,626,582 granted to Muramoto et al. on May6, 1997, teaches a time-based compression method in which two 30 fpsvideo signals are digitized and stored in DRAM memory at a given clockfrequency. Subsequently, the memory is read at twice that writefrequency so that two samples of an original period of 1/30 can beconcatenated in a 1/30 interval. However, depending on the selectedsampling frequency, the final signal will either lack definition becausethe information of two adjacent pixels will be averaged in a singledigital data (low sampling frequency and normal playback frequency), orexceed the capacity of a data storage medium such as a DVD or abroadcast channel. This invention also lacks the ability to generatemultiple output formats from a given source format, and requires twoparallel circuits for reconstruction of the original sequences.

Further, in International application No WO 97/43863, by Briede, laidopen on Nov. 20, 1997, images from a first and a second sequence ofimages are decimated and pixels are redirected to form a single linewith the complementary pixels of two successive original lines and theninterlacing the newly created lines from the left eye and the right eyeto form a combined stereo image sequence to be transmitted through achannel. At the receiving end, the juxtaposed fields are demultiplexedfrom the stereo image sequence and are sent to two parallel expandingcircuits that simultaneously reposition the pixels and recreate themissing picture elements of their respective stereoscopic video sequence(right and left). The thereby reconstructed original first and secondimages sequences are then outputted to two displays for visualization.

While that technology provides an interesting method for spatiallycompressing/decompressing full frames, for storage or distribution usinga limited capacity channel (transport medium), it does not address theproblem of converting a two 24 or 25 fps image sequences into a 30 fpsstereo sequence or boosting the playback rate to prevent flicking.Furthermore, it does not allow playback in other stereoscopic formats,including the page flipping mode using a single display monitor orprojector through time sequencing of the rebuilt first and second imagesequences. Also, as for the previous example, two parallel circuits areagain required to carry out the reconstruction process on both imagesequences since the signal must be originally first demultiplexed beforereconstructing the images.

Although the above examples show that different methods and systems areknown for the encoding of two video signals or images sequences into asingle signal and for decoding such a composite signal to substantiallyretrieve the original signals or sequences, these methods and systems ofthe prior art are nevertheless lacking important features to provide afunctional system which enables high fidelity recording, broadcast andplayback of two 24 fps motion picture movies as well as 25 or 30 fpsstereoscopic video programs, using a single channel and conventionalrecording, playback and display equipment of present manufacture, asrequired for instance to meet the expectations of the home theatremarket for 3D movies reproduction.

There is thus a need for a novel stereoscopic program encoding andplayback method and system which can be readily used with existing hometheatre equipment to provide a high quality stereoscopic reproduction,still at an affordable cost, while enabling playback of a specificstereoscopic video transport signal in a plurality of output formats.

SUMMARY OF THE INVENTION

It is an object of the invention to provide a method for decoding anencoded image stream, said image stream comprising a plurality offrames, each frame consisting of a merged image comprising pixels from aleft image and pixels from a right image, comprising the steps of:

-   -   (a) receiving each merged image;    -   (b) changing a clock domain from the original input signal to an        internal domain;    -   (c) for each merged image, placing at least two adjacent pixels        into an input buffer and interpolating an intermediate pixel,        for forming a reconstructed left frame and a reconstructed right        frame according to provenance of said adjacent pixels; and    -   (d) reconstructing a stereoscopic image stream from said left        and right image frames.

The present invention also provides a system for decoding an encodedimage stream, comprising:

-   -   (a) an input for receiving a compressed image stream;    -   (b) a spatial interpolation module operatively connected to said        input for spatially interpolating each merged frame into a        reconstructed right and left frame;    -   (c) an input memory controller;    -   (d) a right and a left memory buffer for storing said        reconstructed right and left frames;    -   (e) an output memory controller;    -   (f) a rate controller;    -   (g) a synchronisation module for synchronising a stereoscopic        display device; and    -   (h) an output for outputting a stream of reconstructed right and        left images.

More specifically, the system and method of the present inventionincludes the steps of:

-   1—Topologically decimating first and second moving image sequences    having a first display rate, into reduced mosaics to form adjacent    fields of a third image sequence having a second display rate which    is less than twice said first display rate;-   2—Encoding and transporting said third image sequence through a data    transport medium;-   3—Retrieving said third image sequence from said data transport    medium, and,-   4—Topologically interpolating reduced image mosaics from fields of    said third image sequence to substantially rebuild images of said    first and second image sequences.

The process preferably comprises an additional step 5 wherein said firstand second rebuilt image sequences are time sequenced to form an outputimage sequence comprised of successively alternating images from each ofsaid first and second rebuilt image sequences (page flipping mode).According to alternate real time reading modes of the first and secondrebuilt image sequences, output sequence format can be arranged tocomply with a line interleaved, a column interleaved, an anaglyphic or atwo-dimensional presentation mode.

There is further disclosed another embodiment of the process accordingto the present invention wherein step 4 further comprises:

Creating pairs of new images by temporal interpolation of successiveimages of the rebuilt first and second image sequences, and insertingone image from each of said temporally interpolated images pairs intoeach of said first and second rebuilt image sequences to therebyincrease their display rate.

There is further disclosed another embodiment of the process accordingto the present invention wherein step 4 further comprises:

Repeating image pairs into said first and second rebuilt image sequencesto thereby increase their display rate.

There is further disclosed another embodiment of process according tothe present invention wherein, first and second moving image sequenceshave their display rate increased by inserting pairs of new temporallyinterpolated (created) or repeated (read twice) images before carryingout step I of the above process.

According to another embodiment of the present invention, said outputimage sequence has a normal display rate of R images per second and iscomprised of 12 images per second from the first and second imagesequences, 36 images per second spatially interpolated from reducedmosaics of the first and second image sequences, and R-48 insertedimages. Inserted images can be repeated images or temporallyinterpolated images. Should R be equal to 96 images per second, saidoutput image sequence comprises 12 images per second from the first andsecond image sequences, 36 images per second spatially interpolated fromreduced mosaics of the first and second image sequences, each imagebeing repeated twice.

According to a further embodiment of the present invention, said outputimage sequence has a normal display rate of R images per second and iscomprised of 60 images per second spatially interpolated from reducedmosaics of the first and second image sequences, and R-60 insertedimages. Inserted images can be repeated images or temporallyinterpolated images. Should R be equal to 120 images per second, saidoutput image sequence comprises 60 spatially interpolated imagesrepeated twice.

There is further disclosed a system according to the present invention,comprising a digital video disk player, a decoder and a video displaydevice, wherein said decoder:

inputs a signal from the video disk player representative of a sequenceof images having a left field and a right field respectivelyrepresenting reduced image mosaics, topologically interpolates reducedimage mosaics from fields of said image sequence to form a first andsecond sequences of rebuilt images, creates new images by temporalinterpolation of successive images of the rebuilt first and second imagesequences, time sequences rebuilt images and created images to form anoutput image sequence comprised of successively alternating images fromeach of said first and second rebuilt image sequences, and, outputssignals representative of said output image sequence to said videodisplay device.

As it will become more apparent from reading of the following detaileddescription, the present invention overcomes the limitations anddrawbacks of the above mentioned solutions of the prior art, and amongstother advantageous features the following can be underlined:

-   -   The present invention provides an encoding method and a system        implementing the method which enable the compression of two 24        or 30 fps image sequences in a format suitable for storage into        a conventional digital video disk (DVD) or broadcasting using        conventional equipment, and without substantially perceivable        loss of spatial and temporal information.    -   The present invention provides a playback method and system        which enable high visual quality reproduction of stereoscopic        programs by reconstruction of the original sequences and rate        augmentation for presentation at rates from 24 to 60 full        resolution images per second per eye, in progressive or        interlaced mode.    -   The present invention provides an encoding and playback method        and system which features full compatibility with commercially        available data storage medium playback equipment and display        equipment, and more specifically with MPEG main view profile        compression/decompression protocols and commercial circuits.    -   The present invention provides a stereoscopic program playback        method and system which provide universal output signals which        can be directly used or converted to enable reproduction with        any existing technology such as head mounted displays (HMD),        LCD, DLP and CRT rear and front projection TV'S, direct view        TV'S and computer monitors operating under any universal        standard (NTSC, PAL, SECAM, SDTV, HDTV, etc.), with shutter        spectacles, polarized eyewear, or anaglyphic glasses.    -   The present invention provides an encoding and playback method        and system which provides elimination or substantial reduction        of flicking and fatigue usually encountered in viewing        stereoscopic movies with the prior art methods and apparatus.    -   The present invention further provides a method and system which        enables encoding and decoding of two independent image sequences        potentially representing different and unrelated scenes.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1a is a schematic representation of a system according to thepresent invention for the compression encoding of two planar imagesequences into a stereoscopic image sequence to be recorded onto a datastorage medium or broadcast on a single channel.

FIG. 1b is a schematic representation of a system according to thepresent invention, for expansion decoding and playback of a stereoscopicimage sequence previously encoded with a system such as represented inFIG. 1 a.

FIG. 2a is a schematic representation of a portion of a digitized image60 topologically separated into two complementary mosaics of pictureelements, forming fields A and B of a merged image 60.

FIG. 2b is a schematic representation of a portion of two digitizedimages 50 and 50′, topologically decimated into reduced mosaicsrespectively forming field A and field B of a merged image 60.

FIG. 2c is a schematic representation of a process for the spatialinterpolation of a pair of decimated images comprised in a merged image60, to rebuild two full-definition images 72 and 72.

FIG. 2d is a schematic representation of a time-interpolation processfor the creation of a new image 52 from two images 50.4 and 50.5 with atime delay.

FIG. 3a is a schematic representation of a first embodiment of acompression process according to the present invention, for compressionencoding two planar image sequences into a stereoscopic combined imagesequence.

FIG. 3b is a schematic representation of a first embodiment of adecompression process according to the present invention, forreconstruction and temporal expansion of a stereoscopic image sequencepreviously encoded according to a process such as represented in FIG. 3a.

FIG. 4a is a schematic representation of a second embodiment of acompression process according to the present invention, for compressionencoding two planar image sequences into a stereoscopic image sequence.

FIG. 4b is a schematic representation of a second embodiment of acompression process according to the present invention, forreconstruction and temporal expansion of a stereoscopic image sequencepreviously encoded according to a process such as represented in FIG. 4a.

FIG. 5 is a schematic representation of a decoder according to apreferred embodiment of the invention.

Similar reference numerals refer to similar parts throughout the variousFigures.

DETAILED DESCRIPTION OF THE DRAWINGS

Preferred embodiments of the method and associated systems for encodingand playback of stereoscopic video sequences according to the presentinvention will now be described in detail referring to the appendeddrawings.

Referring to FIG. 1, there is illustrated a typical system set-upaccording to the present invention, for the compression encoding of twoplanar image sequences into a stereoscopic image sequence. A first and asecond source of image sequences represented by cameras 3 and 6 arestored into common or respective digital data storage media 4 and 7.Alternatively, image sequences may be provided from digitised moviefilms or any other source of digital picture files stored in a digitaldata storage medium or inputted in real time as a digital video signalsuitable for reading by a microprocessor based system. Cameras 3 and 6are shown in a position wherein their respective captured imagesequences represent different views with a parallax of a scene 100,simulating the perception of a left eye and a right eye of a viewer,according to the concept of stereoscopy. Therefore, appropriatereproduction of the first and second captured image sequences wouldenable a viewer to perceive a three-dimensional view of scene 100.

Stored digital image sequences, typically available in a 24 fps digitalY U V format such as Betacam 4:2:2 (motion pictures), are then convertedto an RGB format by processors such as 5 and 8 and fed to inputs 29 and30 of moving image mixer unit 1, representing the main element of theencoding system of the present invention. It should be noted however thetwo image sequences can alternatively be converted on a time-sharingbasis by a common processor, in order to reduce costs. Mixer 1compresses the two planar RGB input signals into a 30 fps stereo RGBsignal delivered at output 31 and then converted by processor 9 into abetacam 4:2:2 format at output 32 and in turn compressed into a standardMPEG2 bit stream format by a typical circuit 10. The resulting MPEG2coded stereoscopic program can then be recorded on a conventional mediumsuch as a Digital Video Disk (DVD) 11 or broadcasted on a singlestandard channel through, for example, transmitter 13 and antenna 14.Alternative program transport media could be for instance a cabledistribution network or internet.

Turning now to FIG. 1b , there is illustrated a typical system accordingto the present invention for the decoding and playback of thestereoscopic program recorded or broadcasted using the system of FIG. 1.The stereo DVD 11 (3DVD) comprising the compressed information from thefirst and second images sequences, is played by a conventional player 15of current manufacture, delivering a NTSC serial analog signal to theinput 28 of the stereo image decoder 2, the main element of thedecode/playback system of the present invention. Alternatively, any ATSCDTV signal in its analogue or digital format can be accepted.

Decoder 2 produces a synchronized pair of RGB signals at outputs 23 and24, representative of the first an second image sequences, to drive adual input stereoscopic progressive display device such as a headmounted display (HMD) 16. Further, decoder 2 produces a time-sequencedstereo RGB signal at output 25, to supply a single input progressivedisplay device such as projector 17, LCD display 22, CRT monitor or aSDTV or HDTV 21, whereby images from the first and second imagesequences are presented in an alternating page flipping mode.Alternatively, the stereo RGB signal from output 25 may be convertedinto an interlaced NTSC signal to be reproduced by an analog CRTtelevision set or in other stereoscopic formats (ex: column interleavedfor autostereoscopic lenticular displays). Also, decoder 2 may be sointernally configured to output the stereo RGB signal at one of RGBoutputs 23 or 24, thus eliminating output 25.

Decoder 2 further produces a sync-timing signal at output 26 to drive aninfrared shutter spectacle driver 20, driving spectacles 19. Shutterspectacles 19 can be worn by a viewer to view a three-dimensionalprogram projected for instance on screen 18 by projector 17 fed bystereo output 25, by enabling the viewer to alternately see an imagefrom the first image sequence with one eye and an image from the secondimage sequence with his second eye.

As stated in the foregoing description, the two original image sequencescontain too much information to enable direct storage onto aconventional DVD or broadcast through a conventional channel using theMPEG2 or equivalent multiplexing protocol handling information at a rateof 30 fps. Therefore mixer 1 carries out a decimation process to reduceeach picture's information by half.

The spatial decimation carried out by mixer 1 will now be described withreference to FIGS. 2a and 2 b.

FIG. 2a illustrates a portion of an image 50 as defined by a RGB videosignal processed by mixer 1 and decoder 2. As can be seen, image 50 iscomprised of a plurality of pixels (alternating full and empty dots). Inthe RGB format, each pixel is defined by a vector of 3 digital numbersrespectively indicative of the red, green and blue intensity. Thepresent invention makes use of a presumption that three adjacent pixelshave intensities that are not drastically dissimilar, either in thehorizontal direction or the vertical direction. Consequently, thepresent invention advantageously provides for a compression, ordecimation, or separation process which reduces the size of an image by50% without unduly corrupting the image.

In a schematic representation, FIG. 2a illustrates how this is achieved.As mentioned above, image 50 is comprised of a plurality of pixels. Theseries of pixels (indicated by a solid dot) starting with the firstpixel of the image (pixels one of line one), followed by the third oneand so forth throughout the image, from the left to the right of eachrow and from the upper line to the last one are placed in one half of aframe buffer, in order to make mosaic A. The remaining pixels, viz. theeven-numbered pixels, (indicated by the empty dots) are placed as mosaicB. In the example of FIG. 2a , the two complementary mosaics of image 50are shown as being respectively stored in mosaic A and mosaic B of acommon merged image 60. In practice however, this “separation” ispreferably automatically done with appropriate hardware and softwarewhich could, for example, only “read” the odd-numbered or even-numberedpixels and directly placed them in a frame buffer.

As better illustrated in FIG. 2b , basically, images are spatiallycompressed by 50% by keeping only mosaic A of images of the firstsequence (ex. left eye sequence) such as 50, and mosaic B of the imagesof the second sequence (ex. right eye sequence) such as 50′. Keepingmosaics of different types for each sequence promotes higher fidelity atplayback when first and second sequences represent different views of asame scene. Alternatively, spatial compression could be carried out bysaving mosaic A for even numbered images and mosaic B for odd numberedimages, for both input sequences, so that two successive images of thesame eye would be rebuilt from mosaics of different types andpotentially stored in the same compressed frame.

The above operation is accomplished by inputting the data of one pixelat a time in a three-pixel input buffer 55 as shown in FIG. 2b . Pixelinformation is then transferred into the appropriate memory location ofone or more frame buffer(s), each serving to build a different mergedimage. Mosaics from different input images are concatenated side by sideby pair to form two adjacent fields (left field and right field) of anew series of merged frames of the original size such as 60. In theexample illustrated at FIG. 2b , image 50′ is currently being processed,while processing of image 50 is completed, yielding a complete type Amosaic stored in the left field (A) of merged image 60. It should bepointed out however that the merged frames do not necessarily comprisean image from the first sequence and an image from the second sequence,or images captured at the same time, as will be apparent from thedetailed description of preferred embodiments of thecompressing/encoding (mixing) method. As a matter of fact, in theexample of FIG. 2a , field A and field B of the merged image 60 arerespectively filled with mosaic A and mosaic B from a same image 50.While that situation has been chosen to simplify the illustration andcorresponds to an actual situation according to one of the embodimentsof the invention contemplated herein, it shall be deemed that mergedimages such as 60 could comprise mosaics originating from any of theinputted images. That side-by-side compressed transport format is mostlytransparent and unaffected by the compression/decompression processingcharacterizing the MPEG2 main view protocol downstream in the process.

Upon decoding of the merged images, reconstruction of the completeimages is carried out by spatially interpolating missing pixels from thecompressed half-size images (mosaics) located in the fields of themerged images such as 60. As illustrated in FIG. 2c , this is preferablyaccomplished in real time when each pixel of an input merged frame 60decoded in decoder 2 is being transferred to or from memory. Asmentioned above, the underlying premise of the system of the presentinvention is that values of adjacent pixels are not so dissimilar.Consequently, in order to reconstruct an image from a mosaic, adjacentpixels are weighted in order to interpolate a missing pixel.

In a preferred embodiment of the invention, data of one pixel at a timeis stored into a three-pixel input buffer 65. As shown, the three pixelsof the shadowed portion of input image 60 have been stored in inputbuffer 65, two adjacent pixels from the same mosaic being identified asP_(i) and P_(i+1). Data of a third pixel P_(j) is then calculated asbeing the arithmetic mean of each of the 3 components of the RGB vectorsof adjacent pixels (P and P_(i+1)). For example, if pixel P_(i) has anintensity vector of (10,0,30) and pixel P_(i+1) has an intensity vectorof (20,0,60), then, pixel P_(j) will be calculated as being (15,0,45).Therefore, the mean of two identical pixels is another identical pixel.That calculated (topologically interpolated) pixel replaces the missingpixel decimated upon creation of the mosaics from original imagesequences such as 50.

The original pixels and the interpolated pixels are then stored inappropriate memory locations of a frame buffer where the correspondingimage is to be reconstructed (image 72 in the present example). Passedthe centre of each line of the merged frame 60 (entering the rightfield), data is stored into a second frame buffer 72′, to rebuild theimage from the mosaic stored in the right hand field of the stereoimage. The process is followed line by line from left to right, untilthe two images are spatially reconstructed in their respective buffer.

Although the above embodiment interpolates a pixel as being the mean oftwo adjacent pixels of a mosaic, the invention provides for a weightingof more than two pixels. For example, if pixel P_(j) is to beinterpolated, then the two or three preceding and following pixels fromthe mosaic can be used with difference coefficients. More specifically,referring to FIG. 2c , P_(j) can be interpolated as0.6P_(i+1)+0.6P_(i−1)−0.1P_(i+2)−0.1P_(i−2). Of course a variety ofdifferent coefficients and formulae may be used according to preferredresults. Furthermore, instead of performing horizontal interpolation,vertical interpolation may be performed following the same process, or acombination of both horizontal and vertical interpolation.

In order to assure flickerless viewing, the decoding method furthercomprises temporal expansion of image sequences as will be described indetail in the following description. When frame buffers are completelyfilled to provide complete rebuilt or temporally interpolated images (nomore than four frame buffers are required in any embodiment of thereconstruction process and system), they may be read according todifferent modes to provide different types of desired output signals.

A first embodiment of the mixing method carried out by mixer 1 accordingto the present invention is schematically represented in FIG. 3a of theappended drawings and will now be described in detail.

A first sequence of images in RGB 24 fps format 50, identified as L1 toL4, is first time expanded by 25% to form a 30 fps sequence of imagessuch as 51, by the creation and insertion of a new image 52 after everyfourth image of the original sequence 50. New image 52 istime-interpolated from the topological information of the immediatelypreceding and following images (#4 and #5 of original sequence 50). Eachpixel of the new image 52 is calculated as the arithmetic mean of thecorresponding pixel in the precedent and following image, in a mannersimilar to the spatial interpolation technique explained in theforegoing description. FIG. 2d provides a specific illustration of thetime-interpolation process where a new image 52 is created from twotime-successive images 50.4 and 50.5 of input image sequence 50.Creation of new images in the present invention is generallyaccomplished according to that technique to provide improved fluidity atplayback, as compared for instance to simple repetition of frames, whichwould require less processing power.

Alternatively, any known method, such as movement anticipation basedmethods, could be used for performing time-interpolation of images.

Images of the time-expanded sequence 51 are then spatially compressedaccording to the technique illustrated in FIG. 2b and described indetail in the foregoing description, to form mosaics represented by anew sequence 53. Similarly, the second input image sequence 50′ is timeexpanded into the 30 fps sequence 51′ and spatially compressed intomosaics represented by sequence 53′. In this specific embodiment, pairsof compressed images (mosaics) from the first sequence 53 and the secondsequence 53′ respectively are then concatenated to form a left field anda right field of merged images of a 30 fps RGB sequence 60. Thissequence 60 can the be encoded in stream 62 for transmission to a remotelocation.

It is worth mentioning that in spite of the schematic diagram of FIG. 3a, the processing of the image sequences is preferably not carried out inparallel and with long sequences of images. Actually, only a few imagesare being buffered at a given time to enable temporal interpolation, andimages are alternatively imported from the first and the second (leftand right) sequences and processed on a pixel by pixel basis more likethe process steps represented in FIGS. 2b and 2 d.

The decoding and reconstruction carried out by decoder 2 according tothe first embodiment of the present invention will now be described byreferring to FIGS. 3b, 2c and 2 d.

In the example shown in FIG. 3b , stereo MPEG signal 62 is read from aDVD and is converted by the DVD player 15 (FIG. 1b ) into an analog NTSCsignal 70 inputted by the decoder 2. NTSC signal 70 is first convertedinto an RGB format to recuperate merged images such as in sequence 60.Reconstruction of first and second original sequences 50, 50′ can thenbe started by spatially interpolating and separating mosaics from theleft and right fields of the merged images from sequence 60 on a pixelby pixel basis, as previously described with reference to FIG. 2c , toform 30 fps decompressed buffered images such as 72 and 72′. Therefore,spatial interpolation and separation are actually carried outsimultaneously. Sequences of RGB images 72 and 72′ could be directlyoutputted and displayed on a dual input device to reproduce the originalprograms or stereoscopic program signals at a 60 fps (30 per eye)presentation rate. Further processing could also be performed to presentthe image sequences in an interlaced mode or in sequence (page flippingmode), anaglyphic, column interleaved, conventional 2D mode, etc. on aplurality of existing single input display devices.

However, in order to enable comfortable and fatigue free viewing,decoder 2 significantly reduces flicking by providing output signals ata typical rate of 36 full definition frames per eye per second, whilesatisfying results may be obtained at 30 fps per eye with highdefinition frames to match refresh rates of SDTV or HDTV for instance.On the other hand, output signals up to 120 fps (60 images per secondper eye) can be provided by decoder 2 for a very high fidelityreproduction, such an output being compatible however with displaydevices such as DLP projectors and a limited number of high end devices.By experience, a playback rate of 72 fps provides very good results,provided image quality is preserved throughout the coding/decodingprocess as contemplated herein, such a frequency being a standard formost display devices currently encountered in home theatre systems.

Therefore, the playback process carried out by decoder 2 preferablyincludes a further step to increase the presentation rate of sequences72 and 72′. Additional images are inserted at regular intervals in theimage sequence, using the temporal interpolation technique alreadyexplained in the foregoing description of the mixing process referringto FIGS. 3a and 2c . The position of insertion can be carefullycontrolled through the frame number information stored in blank lines ofinput sequence 60 at mixing. Alternatively, images from sequences 72 and72′ can be repeated (read twice) to increase the rate at presentation.For instance every image of the sequences could be read twice to doublethe rate of presentation.

In the example illustrated in FIG. 3b , one new intermediate image pair73, 73′, is time-interpolated using information from images #2 and #3 ofsequences 72 and 72′ respectively and inserted between images #2 and #3to thereby increase the rate of the resulting sequences 74 and 74′ to 36fps (total of 72 fps for the stereoscopic program). The process ispartly illustrated in FIG. 2c , where images #2 and #3 of sequence 72are identified as 72.2 and 72.3. Alternatively, further images could betime-interpolated and inserted to provide a rate of say 48 frames persecond per sequence for a total of 96 fps. A rate of 60 fps per sequence(eye) (total 120 fps for the stereoscopic program) is also aninteresting case where no interpolation is required. All of the imagesof sequences 72 and 72′ are merely duplicated to double the number ofimages. At playback, shutter spectacles are driven at a rate of 120 Hzand all the images of a given sequence are presented twice to thecorresponding eye in 1/30 s. An unsurpassed clarity is thereby provided,but presently only a very limited range of display devices can handlesuch a high refresh rate.

It should be noted that the foregoing description has been based on thefact that input sequences are supplied at a rate of 24 fps, which iscommon for motion picture movies. However, one can easily appreciatethat the mixing process can be easily adapted to the case whereby two 30fps sequences (ex. TV programs) would be supplied, by merely skippingthe preliminary step of temporal interpolation represented bytime-expanded sequences 51 and 51′ of FIG. 3a . Obviously, since thedecoding process always operates on a 30 fps input sequence, nosubstantial adaptation is required to that part of the process.

Alternatively, as illustrated in FIG. 4a , the encoding process of thepresent invention does not require temporal interpolation beforecreating the mosaics. In the example of FIG. 4a , a two 24 fps sequenceis mixed to provide a 30 fps sequence by appropriately separating theframes. Since time-interpolated images may be inserted in the sequence(when input sequences comprise 24 fps), compressed sequence 80 becomesirregular. Therefore, the encoding (mixing) process according to thepresent embodiment of the invention further includes insertion ofinformation in the compressed sequence 60 to enable identification offrame numbers as needed by the reconstruction process to identify imagecontent and rebuild sequences with the proper sequential order (timing)at appropriate locations in the sequence. Such information may be storedin blank lines of merged images for instance. The usefulness of thatprocedure will become more apparent upon reading of the followingdescription of the decoding process. This completes the mixing procedureper se carried out by mixer 1. Further in the process, as described inreference to FIG. 1a , RGB merged image sequence 60 (ex. AVI file) canbe converted to a digital Y U V format prior to being multiplexed intoan MPEG2 bit stream format or be directly converted into an MPEG2 formatidentified by numeral 62 in FIG. 3 a.

FIG. 4b illustrates how the compressed stereo sequence 80 can be decodedto provide two 36 fps streams, using a spatial and temporalinterpolation.

A more specific representation of the decoder 2 of the present inventionis shown in FIG. 5. However, it should be understood that variations arepossible, depending, for example, on whether an all software, allhardware, or mixture of both, solution is chosen.

As can be seen, the decoder has two inputs: an analog and a digital. Ifthe signal is analog, it is converted into a digital signal by ADC 101.FIFO buffer 103 changes the clock domain of the input signal into aclock domain used by the decoder. In practice, a broadcast signal or aDVD signal are clocked at a frequency different from the frequency usedfor RGB signals, hence the necessity of FIFO buffer 103. The signal isthen passed through converter 105 which converts the signal from a YC_(B) C_(R) signal into an RGB signal of 1×720×480 (pixels). This signalis then spatially interpolated according to the teachings of the presentinvention by spatial interpolator 107, resulting is a dual stream of720×480 pixels. This dual stream is then scaled in scaler 109 to providetwo 640×480 image streams (always in the RGB format). Alternatively,other resolutions can be supported by the system of the presentinvention. The frames are then placed in frame buffers 113, one for theright frames and the other for the left frames, the contents of whichare controlled by input memory controller 111.

The output of the frame buffers is controlled by output memorycontroller 115 and, if necessary, time interpolated 117 to increase theframe rate.

A rate controller 119 is preferably provided. The purpose of ratecontroller is to accommodate variations in the clock signals, whichvariations, although minute, de-synchronise the system. Rate controllermonitors the difference in rate and corrects the output frequency byadding or removing a certain number of pixels on inactive lines of theframe. For example, for a frame it may be necessary to add a few pixelsto artificially slow the internal clock and to properly synchronise theclocks.

Another advantageous component of decoder 2 is the anti-flicker filter121. Flicking occurs when wearing shutter spectacles, when there is acontrast between the image and the closing of the shutter of the headdisplay. It has been surprisingly discovered that by evaluating thevalue of the green level in each RGB pixel, and by decreasing thecorresponding pixel colours proportionally when the green level is abovea certain value, flicking is greatly reduced.

The output is then directly digital, or converted into an analog signalby DAC 123. Sync module 125 synchronises the head display with theoutput signal in order to open and close the shutters at the appropriatetimes.

Further preferably, an adjuster 127 is further provided. This adjusteris useful when the display device includes its own frame buffer, whichwould otherwise result in a de-synchronisation between the sync signalfor the shutters and the actual display. This is a manual adjustmentthat the user makes in order to reduce crosstalk/ghosting of the image.

A second embodiment of the mixing method carried out by mixer 1according to the present invention will now be described in detail, byreference to FIG. 4a of the appended drawings. This second embodiment isparticularly advantageous for addressing the problem of converting twoimage sequences available in a 24 fps format to produce a 30 fps MPEG2(main view profile) fully compatible sequence.

Full definition images from the two 24 fps sequences 50 and 50′comprising mosaics A and B by definition are identified as L_(i)AB andR_(i)AB respectively (supposing two sequences of a stereoscopicprogram), index “i” representing the sequential number of a given imageat time t. Dashed lines in FIGS. 4a and 4b indicate frame sequence. In asimilar manner as for the first embodiment previously described, eightinput images are spatially compressed and time-expanded to form five newmerged images in a new 30 fps sequence 80. It should be noted that inthis embodiment of the present invention, 25% more of the original imageinformation is preserved to be recorded or broadcasted. Indeed, two outof the eight original images (images L1 and L2 in the example shown)have both of their mosaics A and B saved in fields of the compressed,sequence 80 instead of one according to the first embodiment.

These fully saved images are nevertheless encoded in the form of twocomplementary mosaics stored in side-by-side merged images fields toascertain homogeneity of the encoded sequence and compatibility with theMPEG2 compression/decompression protocol, by providing a certaintemporal redundancy between successive images. Better definition andfidelity is thus generally obtained at playback with respect to thepreviously described embodiment, but at the expense of increasedprocessing power requirement and system hardware cost. As for theabove-described first embodiment, the encoding (mixing) processaccording to the present embodiment of the invention also furtherincludes insertion of information in the compressed sequence 80 toenable identification of frame numbers as needed by the reconstructionprocess to identify image content and rebuild sequences with the propersequential order and insert interpolated images at appropriate locationsin the sequence. Again, such information may be stored in blank lines ofmerged images for instance.

The corresponding decoding process carried out by decoder 2 according tothe present invention is schematically represented in FIG. 4b andoperates as follows.

The five merged frames 81 to 85 representative of 30 fps RGB inputsequence 80 are expanded to twelve images (six per channel) providingplayback sequences 90 and 100 at 36 fps total (72 total, 36 per eye inthe case of a three-dimensional stereoscopic program). In total, eachgroup of twelve successive images of playback sequences 90 and 100,presented in a page flipping mode according to the frame sequenceindicated by dashed lines 110, comprises two integral original images,six spatially interpolated images and four temporally interpolatedimages. Alternatively, sequences 90 and 100 could be outputtedseparately in parallel on two separate channels, as required by somedisplay devices such as a head mounted or auto-stereoscopic devices. Inthe illustrated example:

-   1. Image 91 (L₁AB) is totally rebuilt from mosaic L₁A stored in the    left field of frame 81 of sequence 80, and mosaic L₁B stored in the    right field thereof;-   2. Image 101 (R₁AX) is spatially interpolated from mosaic R₁A, taken    from the left field of frame 82 of sequence 80;-   3. Image 103 (R₂BX) is spatially interpolated from mosaic R2B, taken    from the right field of frame 82 of sequence 80;-   4. Image 102 is temporally interpolated from image 101 and image    103;-   5. Image 93 (L₂AB) is totally rebuilt from mosaic image L2A, stored    in the left field of frame 83 of sequence 80, and mosaic L₂B stored    in the right field thereof-   6. Image 92 is temporally interpolated from image 91 (L₁AB) and    image 93 (L₂AB);-   7. Image 94 (L₃AX) is spatially interpolated from mosaic L₃A, stored    in the left field of frame 84 of sequence 80;-   8. Image 96 (L₄BX) is spatially interpolated from mosaic L₄B, stored    in the right field of frame 84 of sequence 80;-   9. Image 95 is temporally interpolated from images 94 and 96; Image    104 (R₃AX) is spatially interpolated from mosaic R₃A, stored in the    left field of frame 85 of sequence 80;-   11. Image 106 (R₄BX) is spatially interpolated from mosaic R₄B,    stored in the right field of frame 85 of sequence 80; and-   12. Image 105 is temporally interpolated from image 104 and image    106.

Obviously, one may easily understand that such a reconstruction processrequires proper identification of frame order in the 5 frame sequencesconstituting input sequence 80. Therefore, a frame recognition circuitis provided in decoder 2 to interpret frame number information stored bymixer 1 in merged image sequence 80.

It can be observed that in this latter embodiment as well as in thefirst one disclosed in the foregoing description, the first and secondimage sequences are being encoded and decoded totally independently,without any inference between each other, enabling processing oforiginal video sequences referring to independent scenes.

The above described example of the second embodiment, processing sourcesat 24 fps to yield a presentation rate of 72 fps, is only illustrativeof a more general process applicable to 24 or 30 fps sources to producea stereo output at presentation rates such as 60, 72, 96 or 120 fps. Thechart below provides additional exemplary arrangements for 24 or 30 fpssources and 60, 72, 96 or 120 fps presentation rates:

Spatially- Temporally- Source Output Original interpolated interpolatedRepeated (fps) (fps) images images images images 24 + 24 60 12 36 12 or0 0 or 12 24 + 24 72 12 36 24 0 24 + 24 96 12 36 0 48 24 + 24 120 12 3612 60 30 + 30 60 0 60 0 0 30 + 30 72 0 60 12 or 0 0 or 12 30 + 30 96 060 36 0 30 + 30 120 0 60 0 60

As stated above, RGB sequences 90 and 100 obtained through the abovedescribed processing could be directly outputted and displayed on a dualinput device to reproduce the original programs or stereoscopic programsignals at a 72 fps (36 per eye) presentation rate. Further processingis however carried out by decoder 2 to provide a combined stereoscopicRGB output signal (not shown) comprising images of sequences 90 and 100in a time sequenced arrangement as indicated by dashed arrows such as110. Still referring to the example of FIG. 4b , images would be timesequenced by alternating left eye and right eye images in the followingorder 91, 101, 92, 102, 92, 103, 94, 104, 95, 105, 96, 106. This isaccomplished through an appropriate read sequence of the complete imagesstored in memory buffers.

Presentation of the time sequenced combined signal with a standardprojector or another display device is thus enabled to display thestereoscopic program in a page-flipping mode. Decoder 2 provides thenecessary timing signals to a driver of shutter spectacles which can beworn by a viewer to view the displayed stereoscopic program in athree-dimensional mode, with high fidelity, negligible flicking and highcomfort. As stated above, presentation rate can be increased up to 120fps by inserting additional temporally interpolated image pairs or byrepeating certain image pairs in the decoding process. It is alsocontemplated in the present invention that the RGB combined stereooutput signal could be converted to another known standard presentationformat such as an interlaced format or a conventional 2D format.

Therefore, one can easily appreciate that the above describedembodiments of the present invention provide effective and practicalsolutions for the recording of two motion picture sequences on aconventional data storage medium, and playback with conventionalvideodisk player or broadcast source and display device, to enableviewing of stereoscopic 3D movies at home with unmatched performance andcomfort, still at an affordable cost, in a plurality of output modes tomatch input signal requirement of a broad range of display devices. Forexample a universal set top box fed with a single input signal format asdefined in the foregoing description, can be provided with selectablemodes such as: page flipping, row interleaved, column interleaved,simultaneous dual presentation, anaglyphic, etc. The encoding/playbackmethod and system of the present invention can thus be advantageouslyused in miscellaneous applications, including the processing of videosequences representing independent scenes, with numerous advantages overthe solutions of the prior art.

It will thus be readily appreciated that the present invention presentsadvantages over the prior art. It provides a better quality of images,since no frequency filters are used (low pass or band pass),decompression can be effected in real time with minimal resources, iscompatible with progressive or interlaced systems, both at input andoutput, allows for pause, forward, reverse, slow, etc., and supports allstereoscopic displays presently available.

Although the present invention has been described by means of preferredembodiments thereof, it is contemplated that various modifications maybe made thereto without departing from the spirit and scope of thepresent invention. Accordingly, it is intended that the embodimentdescribed be considered only as illustrative of the present inventionand that the scope thereof should not be limited thereto but bedetermined by reference to the claims hereinafter provided and theirequivalents.

What is claimed is:
 1. A method for playback of image streamscomprising: in a conventional 2D mode: driving a single image stream toa 2D display, in a stereoscopic 3D mode: driving a first image streamfor the left eye and driving a second image stream for the right eye toa head mounted display.
 2. The method of claim 1 further using movementanticipation to perform time-interpolation of images in the stereoscopic3D mode.
 3. The method of claim 2 wherein time-interpolation of imagesin the stereoscopic 3D mode is performed in order to improve fluidity ofplayback as compared to simple repetition of frames.
 4. The method ofclaim 2 wherein the first image stream and the second image stream aregenerated using temporal interpolation to increase frame rate of bothimage streams.
 5. The method of claim 4 wherein variations in the clocksignals are accommodated by a rate controller to avoidde-synchronization.
 6. The method of claim 1 wherein at least one of thefirst image stream and the second image stream is generated from acompressed stereoscopic image stream consisting of frames, each frameconsisting of a merged image comprising a compressed left image and acompressed right image said generating comprising restoring pixels fromat least one of the left compressed image and a right compressed imageby spatial interpolation wherein the columnar position of the restoredpixels alternates for each row position.
 7. An apparatus for playback ofa conventional 2D image stream and for playback of a compressedstereoscopic image stream, said compressed stereoscopic image streamconsisting of frames, each frame consisting of a merged image comprisinga compressed left image and a compressed right image comprising: adecoder for decoding the conventional 2D image stream and for decodingthe compressed stereoscopic image stream, an interpolator forreconstructing a reconstructed image from at least one of the compressedleft image and the compressed right image, a head mounted display. 8.The apparatus of claim 7 further comprising a time-interpolation modulefor performing time-interpolation of images in the stereoscopic 3D modeusing movement anticipation.